Open Case Studies : Mental Health of American Youth

Disclaimer: The purpose of the Open Case Studies project is to demonstrate the use of various data science methods, tools, and software in the context of messy, real-world data. A given case study does not cover all aspects of the research process, is not claiming to be the most appropriate way to analyze a given data set, and should not be used in the context of making policy decisions without external consultation from scientific experts.

This work is licensed under the Creative Commons Attribution-NonCommercial 3.0 (CC BY-NC 3.0) United States License.

To cite this case study please use:

Wright, Carrie, and Ontiveros, Michael and Jager, Leah and Taub, Margaret and Hicks, Stephanie. (2020). https://github.com/opencasestudies/ocs-bp-co2-emissions. Mental Health of American Youth (Version v1.0.0).

Motivation

---

The following papers motivated this case study.

Twenge JM, Cooper AB, Joiner TE, Duffy ME, Binau SG. Age, period, and cohort trends in mood disorder indicators and suicide-related outcomes in a nationally representative dataset, 2005-2017. J Abnorm Psychol.128,3 (2019):185-199. doi:10.1037/abn0000410

Olfson, M., Blanco, C., Wang, S., Laje, G. & Correll, C. U. National Trends in the Mental Health Care of Children, Adolescents, and Adults by Office-Based Physicians. JAMA Psychiatry. 71, 81 (2014):81-90. doi: 10.1001/jamapsychiatry.2013.3074.

The main findings of the first article are:

Rates of major depressive episode in the last year increased 52% 2005–2017 (from 8.7% to 13.2%) among adolescents aged 12 to 17 and 63% 2009–2017 (from 8.1% to 13.2%) among young adults 18–25.

Serious psychological distress in the last month and suicide-related outcomes (suicidal ideation, plans, attempts, and deaths by suicide) in the last year also increased among young adults 18–25 from 2008–2017 (with a 71% increase in serious psychological distress), with less consistent and weaker increases among adults ages 26 and over.

Cultural trends contributing to an increase in mood disorders and suicidal thoughts and behaviors since the mid-2000s, including the rise of electronic communication and digital media and declines in sleep duration, may have had a larger impact on younger people, creating a cohort effect.

While the main findings of the second article are:

Compared with adult mental health care, the mental health care of young people has increased more rapidly.

Between 1995-1998 and 2007-2010, visits resulting in mental disorder diagnoses per 100 population increased significantly faster for youths (from 7.78 to 15.30 visits) than for adults (from 23.23 to 28.48 visits) (interaction: P < .001).

Psychiatrist visits also increased significantly faster for youths (from 2.86 to 5.71 visits).

While depression appear to be on the rise for youths, youths also appear to be seeking more mental health care.

In this case study we will evaluate data related to depression episodes and mental health care to evaluate trends overtime. We will be using data from the National Survey on Drug Use and Health (NSDUH). This data was also used in the first study.

Main Questions

---

Our main questions:

1. How have depression rates in American youth changed since 2002, according to the NSDUH data?
   
2. Do mental health services appear to be reaching more youths? How have rates differed between different youth subgroups (gender, ethnicity)?

Learning Objectives

---

avocado update these!

It may be a good idea to provide a link to Rstudio’s webpage. For the first few months using R, I did not differentiate between R and R Studio. It may be a good distinction to make at least implicitly by providing a link.

In this case study, we will determine the percent of youth in America that have had a major depressive episode in the past year since 2002. We will compare how different youth subgroups have changed over time (by age group (12-13,14-15, and 16-17), gender, ethnicity). We will especially focus on using packages and functions from the Tidyverse, such as rvest. The tidyverse is a library of packages created by RStudio. While some students may be familiar with previous R programming packages, these packages make data science in R especially efficient.

---

We will begin by loading the packages that we will need:

library(here)
library(tidyverse)
library(rvest)

I made some modifications to the table below. The tidyverse package hyperlink referenced readr. I thought this was incorrect. I changed this to the tidyverse website and provided a different description. If this was indeed a typo, it may need to be fixed in other case studies.

Package	Use
here	to easily load and save data
tidyverse	R packages for data science
rvest	to scrape web pages

The first time we use a function, we will use the :: to indicate which package we are using. Unless we have overlapping function names, this is not necessary, but we will include it here to be informative about where the functions we will use come from.

Context

---

According to other sources the rate of suicide has increased for most age groups in the United States over the past decade and a half.

source

While suicide does appear to be increasing amoung youths it also appears to be increasing amoung middle aged adults as well for both females and males.

source

source

According to the CDC:

Since 2008, suicide has ranked as the 10th leading cause of death for all ages in the United States. In 2016, suicide became the second leading cause of death among those aged 10–34 and the fourth leading cause among those aged 35–54.

#### [source]

So although sucide is on the rise for most age groups, sucide is one of the top two contributors to death for youths. Thus this warrents further examination of mental health of American youths.

[source]

If you are having thoughts of suicide, please know that you are not alone. If you are in danger of acting on suicidal thoughts, call 911. For support and resources, call the National Suicide Prevention Lifeline at 1-800-273-8255 or text 741-741 for the Crisis Text Line.

I took this from an article.https://www.theatlantic.com/health/archive/2020/06/why-suicide-rates-among-millennials-are-rising/612943/

If you or someone you know may be struggling with suicidal thoughts, you can call the U.S. National Suicide Prevention Lifeline at 800-273-TALK (8255) any time day or night, or chat online. I thook thos from this article. https://www.usatoday.com/story/news/nation/2020/01/30/u-s-suicide-rate-rose-again-2018-how-can-suicide-prevention-save-lives/4616479002/

covid:https://wellbeingtrust.org/areas-of-focus/policy-and-advocacy/reports/projected-deaths-of-despair-during-covid-19/

Historically, suicide rates were much higher before 1950, however, we are seeing an increase in the last 20 years.

[source]

Besides the US, other countries are also experiencing increased reates of depression in youths. See this report from the World Health Organization about rates of depression in other countries.

Great paper about what may be causing increased dpression - and the caveats of if we actually have increased depression: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3330161/

https://www.nimh.nih.gov/health/publications/teen-depression/index.shtml

According to the National Institute of Mental Health (NIMH):

If you are in crisis and need help, call this toll-free number for the National Suicide Prevention Lifeline (NSPL), available 24 hours a day, every day: 1-800-273-TALK (8255). The service is available to everyone. The deaf and hard of hearing can contact the Lifeline via TTY at 1-800-799-4889. All calls are confidential. You can also visit the Lifeline’s website at www.suicidepreventionlifeline.org.

The Crisis Text Line is another free, confidential resource available 24 hours a day, seven days a week. Text “HOME” to 741741 and a trained crisis counselor will respond to you with support and information over text message. Visit www.crisistextline.org.

Also see here for more information about how to recognise and help youths experiencing symptoms of depression.

Limitations

---

Perhaps “underestimates in the p-values…” is not the correct way to phrase this. I would look for a better way to word this.

Wording for this section should be reviewed.

There are some important considerations regarding this data analysis to keep in mind:

1. We treat sample estimates—estimates of the true population value—as observed values. This produces understimates in the p-values of statistical tests conducted.

2. Furthermore, the sampling mechanism utilized can introduce selection bias in cases where the the sampling methods do not produce a representative sample.

3. Data is collected from human participants; this presents the potential for information bias, as there is the potential that partificipants in the sampling frame may for a variety of reasons report inaccurate information.

What are the data?

---

We will be using data from the National Survey on Drug Use and Health (NSDUH) which is directed by the Substance Abuse and Mental Health Services Administration (SAMHSA), an agency in the U.S. Department of Health and Human Services (DHHS).

This survey started in 1971 and is conducted annualy in all 50 states and the District of Columbia. Approximately 70,000 people (age 12 and up) are interviewed each year about health realted issues. Households are randomly selected and than a professional interviewer visists the addresses and asks one or two of the residents to inverview. The interviewer brings a laptop with them that the participants use to fill out the survey which typically takes an hour to complete. If a participant chooses to particpate they receive $30 in cash. All collected information is confidential and is used for disease surveillance and to guide public policy particuarlly focused on drug and alcohol use as well as mental health. See here for more details about the survey.

This data is made available publicly online on the Substance Abuse & Mental Health Data Archive.

At the website for the survey data, you can see that the results are displayed in many tables. Importantly, there is no obvious way to download the data directly from this particular website.

If one clicks on the TOC botton on the far right upper corner they will be directed to another website, where a large pdf document containing of all of the results can be downloaded.

We are interested in investigating how depression rates have changed and how youths are interacting with mental health services. Thus the following tables are of interest to us are:

Table	Details
Table 11.1A	Settings Where Mental Health Services Were Received in Past Year among Persons Aged 12 to 17: Numbers in Thousands, 2002-2018
Table 11.1B	Settings Where Mental Health Services Were Received in Past Year among Persons Aged 12 to 17: Percentages, 2002-2018
Table 11.2A	Major Depressive Episode in Past Year among Persons Aged 12 to 17, by Demographic Characteristics: Numbers in Thousands, 2004-2018
Table 11.2B	Major Depressive Episode in Past Year among Persons Aged 12 to 17, by Demographic Characteristics: Percentages, 2004-2018
Table 11.3A	Major Depressive Episode with Severe Impairment in Past Year among Persons Aged 12 to 17, by Demographic Characteristics: Numbers in Thousands, 2006-2018
Table 11.3B	Major Depressive Episode with Severe Impairment in Past Year among Persons Aged 12 to 17, by Demographic Characteristics: Percentages, 2006-2018
Table 11.4A	Receipt of Treatment for Depression in Past Year among Persons Aged 12 to 17 with Major Depressive Episode in Past Year, by Demographic Characteristics: Numbers in Thousands, 2004-2018
Table 11.4B	Receipt of Treatment for Depression in Past Year among Persons Aged 12 to 17 with Major Depressive Episode in Past Year, by Demographic Characteristics: Percentages, 2004-2018

Data Import

---

Data is often made available online. Usually, the data we are interested in is made available for download on the page as a delimited text file or an excel file. However, sometimes data is not made available in this manner, such as the NSDUH survey data.

How do we proceed in this scenario?

We can manually copy each cell of data, however, this process is often inefficient, subject to error, and not reproducible. Say we wanted to run an analysis next year on the next years data and it happens to be formatted in the same way.

We can also use R for web scraping.

Web scraping is the process of extracting data from a website.

Basic steps of web scraping

There are two main steps to web scraping:

1. Identify location of data on the webpage that will be scraped

2. Save the webpage element to an object

We accomplish STEP 1 with our web browser.

We accomplish STEP 2 in the R programming environment.

I could not find the animation that I referred to on several occasions.

However, I was able to find the sources that I consulted to create the three step rvest process. They are included below

RStudio

Blog

In this case study we will scrape data from the tables on the NSDUH survey website. This data is available in a large PDF with all the results form the year. However it is not easy to find this PDF and it would be difficult and time consuming to find our tables of interest and to extract the data from the pdf with pdftools. Again, if we instead decided to copy paste the data from the website to another file that we would also need to import, this would not be as efficient or reproducible and might result in errors.

Alternatively, we will use the rvest package to scrape the data directly from the tables on the website. Assuming the data next year would be displayed in a similar manner, this could allow us simply modify our code based on the url for the data next year to run the same analysis on the data easily.

The rvest package can be thought of as the pdftools package for webscraping. Upon pulling the data, additional wrangling will likely be required; but like the pdftools package, rvest streamlines the extraction process.

Steps for scraping tables

The two web scraping steps for these tables can be broken down even further:

1. Identify location of data that will be scraped

• right-click to inspect element (webpage)
• hover pointer over components of element (webpage) until the data has been found
• copy Xpath of data sought

2. Save webpage element to an object in R

• import html code for the webpage
• extract pieces of HTML documents (webpage) using Xpath
• parse the extracted data into a data frame

Below is a animated overview of the process.

---

Now let’s go through each step together:

1) Identify location of data that will be scraped

First, let’s go to the web page with all the tables we are interested in scraping

Once on the webpage, there aren’t any visible options to download the data.

Right-click and select “Inspect”

A window opens.

This window allows us to glance at the internal mechanics of the webpage. To scrape the data from the webpage, we need to first learn a little bit about the components that make it the web page it is.

Hovering our mouse over the elements of the webpage highlights the respective section of the webpage it represents. By hovering over several elements—and clicking on the elements on the right side of the screen—we can indentify the element that contains the data we are looking for. Another option for identifying xpaths is to use the selectorgadget tool.

Right click on the element and copy the XPath. We will need this XPath for the next step.

Now we can return to the R programming environment.

---

2) Save webpage element to an object in R

For the first table we want to scrape, the XPath is /html/body/div[4]/div[1]/table. We use this Xpath with functions from the rvest package to scrape the data from this table.

Let’s explore this step in greater detail:

We need to:

• import html code for the webpage
• extract pieces (table) out of HTML documents (webpage) using Xpath
• parse the html table into a data frame

To do this:

• We import the html code using the read_html() function of the rvest package
• We extract specific components of the webpage using the html_nodes() function of the rvest package
• We convert this html table into a dataframe using the html_table()function of the rvest package

The rvest package provides wrappers for the xml2 and httr packages, thus we can just install and load the rvest package and it will install and load dependency packages like xml2 and httr and allow us to use functions from both of these packages.

In fact, when we load rvest the first time we see:

knitr::include_graphics(here::here("img", "rvest.png"))

In this case, we are scraping table 11.1a from the website. First we assign the url to a character string to use within the read_html() function of the xml2 package.

NSDUH_url <- "https://www.samhsa.gov/data/sites/default/files/cbhsq-reports/NSDUHDetailedTabs2018R2/NSDUHDetTabsSect11pe2018.htm"

One could also directly use the url but this is less convenient for piping.

Click here if you are unfamiliar with piping in R, which uses this %>% opperator

By piping we mean using the %>% pipe operator which are usable when loading the tidyverse or several of the packages within the tidyverse like dplyr becuase they load the magrittr package. This allows us to perform multiple sequential steps on one data input.

The read_html() function then allows us to save the html document for the webpage inside R.

webpage <- NSDUH_url %>%
  xml2::read_html() 
webpage

{html_document}
<html lang="en">
[1] <head>\n<!-- Google Tag Manager --><script>(function(w,d,s,l,i){w[l]=w[l] ...
[2] <body>\r\n<!-- Google Tag Manager (noscript) -->\r\n<noscript><iframe src ...

Then we use the html_nodes() function of the rvest package to select just the table11.1a element of the webpage.

See this tutorial (and the answers in case you get stuck) on CSS selectors to understand more about how this function works to use the xpath to select the elements of interest from the webpage.

webpage_element <-webpage %>%
  rvest::html_nodes(xpath='/html/body/div[4]/div[1]/table')
webpage_element 

{xml_nodeset (1)}
[1] <table class="rti">\n<caption>Table 11.1A – Settings Where Mental Health  ...

Finally, the html_table() function of the rvest package parses the html object into a data frame.

library(dplyr)
table11.1a<-webpage_element%>%
  rvest::html_table()
print(table11.1a, max = 2)

[[1]]
     Setting Where Mental Health ServiceWas Received 2002 2003 2004 2005 2006
     2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
 [ reached 'max' / getOption("max.print") -- omitted 21 rows ]

glimpse(table11.1a)

List of 1
 $ :'data.frame':   21 obs. of  18 variables:
  ..$ Setting Where Mental Health ServiceWas Received: chr [1:21] "SPECIALTY MENTAL HEALTH SERVICE1" "Outpatient" "Private Therapist, Psychologist,\r\n   Psychiatrist, Social Worker, or\r\n   Counselor" "Mental Health Clinic or Center" ...
  ..$ 2002                                           : chr [1:21] "2,898a" "2,662a" "2,254a" "611a" ...
  ..$ 2003                                           : chr [1:21] "3,065a" "2,795a" "2,347a" "635a" ...
  ..$ 2004                                           : chr [1:21] "3,348a" "3,015a" "2,523a" "716a" ...
  ..$ 2005                                           : chr [1:21] "3,362a" "3,048a" "2,573a" "657a" ...
  ..$ 2006                                           : chr [1:21] "3,255a" "2,931a" "2,416a" "587a" ...
  ..$ 2007                                           : chr [1:21] "3,104a" "2,787a" "2,365a" "583a" ...
  ..$ 2008                                           : chr [1:21] "3,129a" "2,837a" "2,408a" "567a" ...
  ..$ 2009                                           : chr [1:21] "2,925a" "2,650a" "2,296a" "537a" ...
  ..$ 2010                                           : chr [1:21] "2,920a" "2,635a" "2,265a" "547a" ...
  ..$ 2011                                           : chr [1:21] "3,101a" "2,842a" "2,409a" "547a" ...
  ..$ 2012                                           : chr [1:21] "3,118a" "2,846a" "2,427a" "610a" ...
  ..$ 2013                                           : chr [1:21] "3,341a" "3,064a" "2,572a" "731a" ...
  ..$ 2014                                           : chr [1:21] "3,369a" "3,110a" "2,698a" "760a" ...
  ..$ 2015                                           : chr [1:21] "3,253a" "2,958a" "2,532a" "792a" ...
  ..$ 2016                                           : chr [1:21] "3,598a" "3,239a" "2,819a" "929" ...
  ..$ 2017                                           : chr [1:21] "3,646a" "3,328" "2,908" "995" ...
  ..$ 2018                                           : chr [1:21] "3,901" "3,547" "3,080" "977" ...

We can see that the output is a list with one element, to extract the data from the list we will use brackets [[]] to select the first element of the list.

table11.1a <- table11.1a[[1]]

Putting this all of this together we can do the entire porcess like this with our pipe opperator %>%.

NSDUH_url <- "https://www.samhsa.gov/data/sites/default/files/cbhsq-reports/NSDUHDetailedTabs2018R2/NSDUHDetTabsSect11pe2018.htm"
table11.1a <- NSDUH_url %>%
  xml2::read_html() %>%
  rvest::html_nodes(xpath='/html/body/div[4]/div[1]/table') %>%
  rvest::html_table()
table11.1a <- table11.1a[[1]]

Now need to repeat the above process for the other tables we are interested in.

Writing a function to scrape multiple tables

We can create a function to accomplish this succinctly. Functions allow us to perform the same process on multiple data inputs. See this other case study for more details about how to write a function.

In general, the process pf writing functions involves first specifying an input that is used within the function to create an output. In this case the data input is XPATH which will be replaced by an actual xpath and the nused in the subsequent steps to scrape the data from each table that an xpath is supplied for.

We will all this function scarper.

scraper <- function(XPATH){
  NSDUH_url <- "https://www.samhsa.gov/data/sites/default/files/cbhsq-reports/NSDUHDetailedTabs2018R2/NSDUHDetTabsSect11pe2018.htm"
  table <- NSDUH_url %>%
  read_html() %>%
  html_nodes(xpath=XPATH) %>%
  html_table()
  output <- table[[1]]
  output
}

Now we can apply the function we created to each of the xpaths for each of the tables on the website that we would like to use in our data analysis.

table11.1b <- scraper(XPATH = "/html/body/div[4]/div[2]/table")
table11.2a <- scraper(XPATH = '/html/body/div[4]/div[3]/table')
table11.2b <- scraper(XPATH = '/html/body/div[4]/div[4]/table')
table11.3a <- scraper(XPATH = '/html/body/div[4]/div[5]/table')
table11.3b <- scraper(XPATH = '/html/body/div[4]/div[6]/table')
table11.4a <- scraper(XPATH = '/html/body/div[4]/div[7]/table')
table11.4b <- scraper(XPATH = '/html/body/div[4]/div[8]/table')

Great! We have successfully scraped the data.

Now we need to wrangle the data.

Data Exploration and Wrangling

---

Now that we’ve imported the data, let’s see if we can wrangle a table. Since the data appears to be formated in a similar way in each of the tables, it is likely that whatever steps we take to wrangle this first table will also be necessary in the wrangling of subsequent tables. This is because well-maintained data sources often format different datasets similarly. We can take advantage of this similarity to speed up the wrangling process.

Table11.1a

First we want to remove the last row of our data frame, which happens to be the legend of our table. Recall from looking at the website that there is a legend for this table.

We can take a look at the last row using the tail function of the dplyr package. We can specify that we only want to see the last row by using the n = 1 argument.

table11.1a %>%
  dplyr::as_tibble() %>%
  tail(n = 1)

# A tibble: 1 x 18
  `Setting Where … `2002` `2003` `2004` `2005` `2006` `2007` `2008` `2009`
  <chr>            <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr> 
1 "* = low precis… "* = … "* = … "* = … "* = … "* = … "* = … "* = … "* = …
# … with 9 more variables: `2010` <chr>, `2011` <chr>, `2012` <chr>,
#   `2013` <chr>, `2014` <chr>, `2015` <chr>, `2016` <chr>, `2017` <chr>,
#   `2018` <chr>

We can see that the legend is repeated for every column. Let’s take a look at the year 2004 column.

table11.1a %>%
  dplyr::as_tibble() %>%  
  select(`2004`) %>%
  tail(n = 1)

# A tibble: 1 x 1
  `2004`                                                                        
  <chr>                                                                         
1 "* = low precision; -- = not available; da = does not apply; nc = not compara…

Let’s save this so that we can refer back to it later:

legend <- table11.1a %>%
  as_tibble() %>%  
  select(`2004`) %>%
  tail(n = 1)

Another way to look at the last row is to use the n() function of the dplyr package. This function can be used inside other dplyr functions and it counts the total number of observations of a group. Within the slice() function of the dplyr packge, it allows you to refer the full length of the object.

table11.1a %>%
  dplyr::as_tibble() %>%
  slice(n()) 

# A tibble: 1 x 18
  `Setting Where … `2002` `2003` `2004` `2005` `2006` `2007` `2008` `2009`
  <chr>            <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr> 
1 "* = low precis… "* = … "* = … "* = … "* = … "* = … "* = … "* = … "* = …
# … with 9 more variables: `2010` <chr>, `2011` <chr>, `2012` <chr>,
#   `2013` <chr>, `2014` <chr>, `2015` <chr>, `2016` <chr>, `2017` <chr>,
#   `2018` <chr>

We can use the slice() function of the dplyr package to remove this row, by using the slicefunction to select from the first row using 1: to the second to last row using n()-1.

We are also going to use a special pipe opperator from the magrittr package called the compound assignment pipe-operator or sometimes the double pipe operator. This allows us to use the table11.1a as our input and reasign it at the end after all the steps have been performed.

library(magrittr)
table11.1a %<>%
  dplyr::as_tibble() %>%
  slice(1:(n()-1))

Now let’s take a look at data:

table11.1a

# A tibble: 20 x 18
   `Setting Where … `2002` `2003` `2004` `2005` `2006` `2007` `2008` `2009`
   <chr>            <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr> 
 1 "SPECIALTY MENT… "2,89… "3,06… "3,34… "3,36… "3,25… "3,10… "3,12… "2,92…
 2 "Outpatient"     "2,66… "2,79… "3,01… "3,04… "2,93… "2,78… "2,83… "2,65…
 3 "Private Therap… "2,25… "2,34… "2,52… "2,57… "2,41… "2,36… "2,40… "2,29…
 4 "Mental Health … "611a" "635a" "716a" "657a" "587a" "583a" "567a" "537a"
 5 "Partial Day Ho… "440"  "425"  "439"  "449"  "471"  "416"  "374a" "340a"
 6 "In-Home Therap… "693a" "656a" "762a" "731a" "719a" "707a" "716a" "657a"
 7 "Inpatient or R… "509a" "542a" "629"  "619"  "596"  "581a" "539a" "524a"
 8 "Hospital"       "422a" "467a" "515"  "529"  "516"  "511a" "469a" "440a"
 9 "Residential Tr… "224a" "233a" "299"  "229a" "225a" "199a" "198a" "213a"
10 "NONSPECIALTY M… "nc"   "nc"   "nc"   "nc"   "nc"   "nc"   "nc"   "3,43…
11 "Education2,3"   "nc"   "nc"   "nc"   "nc"   "nc"   "nc"   "nc"   "2,93…
12 "School Social … "nc"   "nc"   "nc"   "nc"   "nc"   "nc"   "nc"   "2,28…
13 "Special School… "nc"   "nc"   "nc"   "nc"   "nc"   "nc"   "nc"   "975a"
14 "General Medici… ""     ""     ""     ""     ""     ""     ""     ""    
15 "Pediatrician o… "657a" "732"  "840"  "810"  "694"  "692"  "710"  "605a"
16 "Juvenile Justi… ""     ""     ""     ""     ""     ""     ""     ""    
17 "Juvenile Deten… "--"   "--"   "--"   "--"   "--"   "--"   "--"   "109a"
18 "Child Welfare"  ""     ""     ""     ""     ""     ""     ""     ""    
19 "Foster Care or… "157a" "179a" "158a" "143a" "129"  "114"  "118"  "92"  
20 "SPECIALTY MENT… "nc"   "nc"   "nc"   "nc"   "nc"   "nc"   "nc"   "1,22…
# … with 9 more variables: `2010` <chr>, `2011` <chr>, `2012` <chr>,
#   `2013` <chr>, `2014` <chr>, `2015` <chr>, `2016` <chr>, `2017` <chr>,
#   `2018` <chr>

Great! We can see the the legend is no longer part of the data.

Now let’s use the legend to recode the data. There are many different values for missing data, that we would like to replace with NA instead. We can use the pull() function of the dplyr package to take a look at the legend data:

pull(legend, `2004`)

[1] "* = low precision; -- = not available; da = does not apply; nc = not comparable due to methodological changes; nr = not reported due to measurement issues.\r\nNOTE: Some 2006 to 2010 estimates may differ from previously published estimates due to updates (see Section 3.3.5 in Chapter 3 of the 2018 National Survey on Drug Use and Health: Methodological Summary and Definitions).\r\nNOTE: Mental health services for persons aged 12 to 17 includes treatment/counseling for emotional or behavioral problems not caused by drug or alcohol use. Respondents with unknown mental health service information were excluded.\r\nNOTE: Respondents could indicate multiple service settings; thus, the response categories are not mutually exclusive.a The difference between this estimate and the 2018 estimate is statistically significant at the .05 level. Rounding may make the estimates appear identical.1 Because of revisions in 2013 to Specialty Mental Health Service estimates, these 2002 to 2012 estimates may differ from estimates published prior to the 2013 NSDUH.2 Because of revisions in 2009 to the questions on the Source of Youth Mental Health Education Services, these estimates are not comparable with the education services estimates published prior to the 2009 NSDUH.3 Respondents who did not report their school enrollment status, who reported not being enrolled in school in the past 12 months, or who reported being home-schooled were not asked about receipt of mental health services from this setting; however, respondents who reported not being enrolled in school in the past 12 months were classified as not having received treatment/counseling from this setting.\r\nDefinitions: Measures and terms are defined in Appendix A.\r\nSource: SAMHSA, Center for Behavioral Health Statistics and Quality, National Survey on Drug Use and Health, 2002-2018."

Looks like we want to replace values that are: *, --, da, nc, and nr. We can use the na_if() function to recode these values to NA.

avocado… there isnt support for doing this in one command… but could at least do two commands

table11.1a %<>%
  dplyr::na_if("nc") %>%
  dplyr::na_if("--") %>%
  dplyr::na_if("") %>%
  dplyr::na_if("*")

table11.1a 

# A tibble: 20 x 18
   `Setting Where … `2002` `2003` `2004` `2005` `2006` `2007` `2008` `2009`
   <chr>            <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr> 
 1 "SPECIALTY MENT… 2,898a 3,065a 3,348a 3,362a 3,255a 3,104a 3,129a 2,925a
 2 "Outpatient"     2,662a 2,795a 3,015a 3,048a 2,931a 2,787a 2,837a 2,650a
 3 "Private Therap… 2,254a 2,347a 2,523a 2,573a 2,416a 2,365a 2,408a 2,296a
 4 "Mental Health … 611a   635a   716a   657a   587a   583a   567a   537a  
 5 "Partial Day Ho… 440    425    439    449    471    416    374a   340a  
 6 "In-Home Therap… 693a   656a   762a   731a   719a   707a   716a   657a  
 7 "Inpatient or R… 509a   542a   629    619    596    581a   539a   524a  
 8 "Hospital"       422a   467a   515    529    516    511a   469a   440a  
 9 "Residential Tr… 224a   233a   299    229a   225a   199a   198a   213a  
10 "NONSPECIALTY M… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   3,430a
11 "Education2,3"   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   2,931a
12 "School Social … <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   2,286 
13 "Special School… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   975a  
14 "General Medici… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>  
15 "Pediatrician o… 657a   732    840    810    694    692    710    605a  
16 "Juvenile Justi… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>  
17 "Juvenile Deten… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   109a  
18 "Child Welfare"  <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>  
19 "Foster Care or… 157a   179a   158a   143a   129    114    118    92    
20 "SPECIALTY MENT… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   1,226a
# … with 9 more variables: `2010` <chr>, `2011` <chr>, `2012` <chr>,
#   `2013` <chr>, `2014` <chr>, `2015` <chr>, `2016` <chr>, `2017` <chr>,
#   `2018` <chr>

Now let’s rename the first column using the rename() function of the dplyr package. This requires listing the new name first like so: new_name = old_name.

table11.1a %<>%
  dplyr::rename(MHS_setting = 
                  `Setting Where Mental Health ServiceWas Received`)
head(table11.1a)

# A tibble: 6 x 18
  MHS_setting `2002` `2003` `2004` `2005` `2006` `2007` `2008` `2009` `2010`
  <chr>       <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr> 
1 "SPECIALTY… 2,898a 3,065a 3,348a 3,362a 3,255a 3,104a 3,129a 2,925a 2,920a
2 "Outpatien… 2,662a 2,795a 3,015a 3,048a 2,931a 2,787a 2,837a 2,650a 2,635a
3 "Private T… 2,254a 2,347a 2,523a 2,573a 2,416a 2,365a 2,408a 2,296a 2,265a
4 "Mental He… 611a   635a   716a   657a   587a   583a   567a   537a   547a  
5 "Partial D… 440    425    439    449    471    416    374a   340a   362a  
6 "In-Home T… 693a   656a   762a   731a   719a   707a   716a   657a   674a  
# … with 8 more variables: `2011` <chr>, `2012` <chr>, `2013` <chr>,
#   `2014` <chr>, `2015` <chr>, `2016` <chr>, `2017` <chr>, `2018` <chr>

Nice!

Now you may notice that the individual values have an "a" after the numeric value.

According to the legend this indicates if “the difference between this estimate and the 2018 estimate is significant at the .05 level.”

While this is useful information, it makes it difficult to work with our numeric values, so we want to remove them.

Since lower case “a” values occur in the first column values, we want to makes sure we dont remove these.

So how can we do this? We can use the stringr package to modify character strings. and we can use the dplyr functions mutate(), select() and across() to specify want columns we want to change.

Currently all of our data is of class character as indicated by the <chr> under the column names.

Click here for an explanation of what a character string is

There are several classes of data in R programming. Character is one of these classes. A chacter string is an individual data value made up of characters. This can be a paragraph, like the legend for the table, or it can be a single letter or number like the letter "a" or the number "3". If data is of class character, than the numeric values will not be processed like a numeric value in a mathematical sense. If you want your numeric values to be interpreted that way, they need to be converted to a numeric class. The options typically used are integer (which has no decimal place) and double precision (which has a decimal place).

The stringr package has functions that allow us to replace (the str_replace() function) or remove(the str_remove() function) characters.

To use these we need to be able to specify what we want to remove and replace.

Here is a part of a cheatsheet about string manipulation from rstudio.

knitr::include_graphics(here::here("img", "regex.png"))

We can see that we can refer to any digit (such as 1,2,3 etc.) as [:digit:]. We can also see that we can refer to any punction mark as [:punct:]. Finally, we see that spaces and tabls can be refered to as [:blank:].

If we take a closer look at the first column of our table (using the pull() function of the dplyr package), we can see that besides the "a" values that we see adjacent to our numeric values in the body of the table, we also some large white spaces, some numeric values, instances of \r\n, as well as some commas and other punctuation marks.

pull(table11.1a, MHS_setting)

 [1] "SPECIALTY MENTAL HEALTH SERVICE1"                                                                                
 [2] "Outpatient"                                                                                                      
 [3] "Private Therapist, Psychologist,\r\n   Psychiatrist, Social Worker, or\r\n   Counselor"                          
 [4] "Mental Health Clinic or Center"                                                                                  
 [5] "Partial Day Hospital or Day Treatment\r\n   Program"                                                             
 [6] "In-Home Therapist, Counselor, or Family\r\n   Preservation Worker"                                               
 [7] "Inpatient or Residential1"                                                                                       
 [8] "Hospital"                                                                                                        
 [9] "Residential Treatment Center"                                                                                    
[10] "NONSPECIALTY MENTAL HEALTH\r\nSERVICE2"                                                                          
[11] "Education2,3"                                                                                                    
[12] "School Social Worker, School Psychologist,\r\n   or School Counselor"                                            
[13] "Special School or Program within a Regular\r\n   School for Students with Emotional or\r\n   Behavioral Problems"
[14] "General Medicine"                                                                                                
[15] "Pediatrician or Other Family Doctor"                                                                             
[16] "Juvenile Justice"                                                                                                
[17] "Juvenile Detention Center, Prison, or Jail"                                                                      
[18] "Child Welfare"                                                                                                   
[19] "Foster Care or Therapeutic Foster Care"                                                                          
[20] "SPECIALTY MENTAL HEALTH SERVICES\r\nAND EDUCATION, GENERAL MEDICINE\r\nOR CHILD WELFARE SERVICES1,2,3"           

We can use the str_remove_all() function which is a variant of the str_remove() function of the stringr package (which allows us to remove all occurences of sepcificed characters in each row rather than just the first occurance, which is what str_remove() does), to remove the digit values, the \r\n characters and the punctionation marks from the column called MHS_setting.

Using the mutate() function we specify that we want to change this particular column and replace it with a version of this column that has removed characters that match digits, r\n or punctionation marks.

We need to specify that the character strings that should be used can be found ing the MHS_setting column by using the string = argument and the patterns to find and remove are specified using the pattern = argument.

To allow us to look for all three of these patterns at the same time, we can use the | symbol between each pattern.

table11.1a %<>%
mutate(MHS_setting = 
         str_remove_all(string = MHS_setting, 
                       pattern = "[:digit:]|\r\n|[:punct:]|"))

We also want to replace the spaces with an underscore. We can see that sometimes there appears to be more than one space. We can specify that we want any occurance of 1 or more to be replaced by using the {1,} notation.

See here for an explanation of this on the cheat sheet:

knitr::include_graphics(here::here("img", "quantifiers.png"))

So now we will use the str_replace_all() function of the stringr package. In this case we also need to specify a replacement with the replacement = argument.

table11.1a%<>%
mutate(MHS_setting = 
         str_replace_all(string = MHS_setting,
                        pattern = "[:blank:]{1,}", 
                    replacement = "_"))

Now to finally remove the “a” values and the commas from the body of the table we can use str_remove_all() function yet again. However this time to specify that we want all columns except the first column called MHS_setting, we can use the across() function of the dplyr package. This allows us to specify what columns we want to mutate by using the .cols = argument. We can select all columns except the first column called MHS_setting by using a minus sign - in front of the column name.

table11.1a%<>%
mutate(dplyr::across(.cols = -MHS_setting,
                stringr::str_remove_all, "a|,"))

table11.1a

# A tibble: 20 x 18
   MHS_setting `2002` `2003` `2004` `2005` `2006` `2007` `2008` `2009` `2010`
   <chr>       <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr> 
 1 SPECIALTY_… 2898   3065   3348   3362   3255   3104   3129   2925   2920  
 2 Outpatient  2662   2795   3015   3048   2931   2787   2837   2650   2635  
 3 Private_Th… 2254   2347   2523   2573   2416   2365   2408   2296   2265  
 4 Mental_Hea… 611    635    716    657    587    583    567    537    547   
 5 Partial_Da… 440    425    439    449    471    416    374    340    362   
 6 InHome_The… 693    656    762    731    719    707    716    657    674   
 7 Inpatient_… 509    542    629    619    596    581    539    524    531   
 8 Hospital    422    467    515    529    516    511    469    440    447   
 9 Residentia… 224    233    299    229    225    199    198    213    217   
10 NONSPECIAL… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   3430   3465  
11 Education   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   2931   2957  
12 School_Soc… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   2286   2214  
13 Special_Sc… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   975    1054  
14 General_Me… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>  
15 Pediatrici… 657    732    840    810    694    692    710    605    601   
16 Juvenile_J… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>  
17 Juvenile_D… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   109    80    
18 Child_Welf… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>  
19 Foster_Car… 157    179    158    143    129    114    118    92     108   
20 SPECIALTY_… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   1226   1283  
# … with 8 more variables: `2011` <chr>, `2012` <chr>, `2013` <chr>,
#   `2014` <chr>, `2015` <chr>, `2016` <chr>, `2017` <chr>, `2018` <chr>

Our table is looking much better!

We would also like to remove a couple of rows that are completely empty. These are the rows where the first column values are Genearl_Medicine and Juvenile_Justice. If we look at the website, we can see that these were leading line with no data.

knitr::include_graphics(here::here("img", "table11.1a.png"))

We can remove them using the filter() function of the dplyr package. We can specify that we dont want to keep these rows by using the != not equal to opperator.

table11.1a %<>%
  dplyr::filter(MHS_setting != "General_Medicine") %>%
  dplyr::filter(MHS_setting != "Juvenile_Justice") 
table11.1a 

# A tibble: 18 x 18
   MHS_setting `2002` `2003` `2004` `2005` `2006` `2007` `2008` `2009` `2010`
   <chr>       <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr>  <chr> 
 1 SPECIALTY_… 2898   3065   3348   3362   3255   3104   3129   2925   2920  
 2 Outpatient  2662   2795   3015   3048   2931   2787   2837   2650   2635  
 3 Private_Th… 2254   2347   2523   2573   2416   2365   2408   2296   2265  
 4 Mental_Hea… 611    635    716    657    587    583    567    537    547   
 5 Partial_Da… 440    425    439    449    471    416    374    340    362   
 6 InHome_The… 693    656    762    731    719    707    716    657    674   
 7 Inpatient_… 509    542    629    619    596    581    539    524    531   
 8 Hospital    422    467    515    529    516    511    469    440    447   
 9 Residentia… 224    233    299    229    225    199    198    213    217   
10 NONSPECIAL… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   3430   3465  
11 Education   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   2931   2957  
12 School_Soc… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   2286   2214  
13 Special_Sc… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   975    1054  
14 Pediatrici… 657    732    840    810    694    692    710    605    601   
15 Juvenile_D… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   109    80    
16 Child_Welf… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>  
17 Foster_Car… 157    179    158    143    129    114    118    92     108   
18 SPECIALTY_… <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   <NA>   1226   1283  
# … with 8 more variables: `2011` <chr>, `2012` <chr>, `2013` <chr>,
#   `2014` <chr>, `2015` <chr>, `2016` <chr>, `2017` <chr>, `2018` <chr>

Now we would like to change the shape of our table so that we have a new column that reperesents the year and a new column that represents the value for that year. To do so we will be making our table “longer”, meaning that it will have fewer rows and more columns. See here for more information about different table formats, typically refreed to as wide and long or sometimes narrow.

We will use the pivot_longer() function of the tidyr package to change the shape of our table.

There are 3 main arguments in this function:
1) cols - which specifies what columns to collapse
2) names_to - which specifies the name of the new column that will be created that will contain the column names of the columns you are collapsing
3) values_to - which specifies the name of the new column that will be created that will contain the values from the columns you are collapsing

To specify that we want to collapse all columns except the MHS_setting column we can again use the minus sign.

library(tidyr)
table11.1a %<>%
  tidyr::pivot_longer(cols = - MHS_setting, 
                  names_to = "Year",
                 values_to = "Number")
table11.1a

# A tibble: 306 x 3
   MHS_setting                     Year  Number
   <chr>                           <chr> <chr> 
 1 SPECIALTY_MENTAL_HEALTH_SERVICE 2002  2898  
 2 SPECIALTY_MENTAL_HEALTH_SERVICE 2003  3065  
 3 SPECIALTY_MENTAL_HEALTH_SERVICE 2004  3348  
 4 SPECIALTY_MENTAL_HEALTH_SERVICE 2005  3362  
 5 SPECIALTY_MENTAL_HEALTH_SERVICE 2006  3255  
 6 SPECIALTY_MENTAL_HEALTH_SERVICE 2007  3104  
 7 SPECIALTY_MENTAL_HEALTH_SERVICE 2008  3129  
 8 SPECIALTY_MENTAL_HEALTH_SERVICE 2009  2925  
 9 SPECIALTY_MENTAL_HEALTH_SERVICE 2010  2920  
10 SPECIALTY_MENTAL_HEALTH_SERVICE 2011  3101  
# … with 296 more rows

We can see that our table is now much longer- as we have 340 rows!

Question Opportunity

Why do we have 340 rows now?

Finally, we also want to change our values to be numeric as opposed to character so that we can use them in mathematical functions. We can use the base as.numeric() function. Again we will use the across() function to indicate what varaibles we wish to mutate.

table11.1a %<>%
  mutate(across(.cols =c(Year, Number), as.numeric))

table11.1a

# A tibble: 306 x 3
   MHS_setting                      Year Number
   <chr>                           <dbl>  <dbl>
 1 SPECIALTY_MENTAL_HEALTH_SERVICE  2002   2898
 2 SPECIALTY_MENTAL_HEALTH_SERVICE  2003   3065
 3 SPECIALTY_MENTAL_HEALTH_SERVICE  2004   3348
 4 SPECIALTY_MENTAL_HEALTH_SERVICE  2005   3362
 5 SPECIALTY_MENTAL_HEALTH_SERVICE  2006   3255
 6 SPECIALTY_MENTAL_HEALTH_SERVICE  2007   3104
 7 SPECIALTY_MENTAL_HEALTH_SERVICE  2008   3129
 8 SPECIALTY_MENTAL_HEALTH_SERVICE  2009   2925
 9 SPECIALTY_MENTAL_HEALTH_SERVICE  2010   2920
10 SPECIALTY_MENTAL_HEALTH_SERVICE  2011   3101
# … with 296 more rows

Now we see that the Year and Number variables are of class double because of the <dbl> under the column name.

Now we want to wrangle all of our other tables which are formatted similarly.

We can again write a function to simplify this process.

The function needs to:

• remove the last row of the table
• get rid of certain patterns
• transition the data to long format

data_prep_settings <- function(TABLE, old_col, new_col, pivot_col){
  TABLE <- TABLE[-dim(TABLE)[1],]
  TABLE <- TABLE %>%
  na_if("nc") %>%
  na_if("--") %>%
  na_if("") %>%
  na_if("*")
  TABLE <- TABLE %>%
    as_tibble() %>%
    rename({{new_col}} := {{old_col}})
  partA <- TABLE %>%
    select({{new_col}})
  partB <- TABLE %>%
    select(-{{new_col}})
  partA <- partA %>%
  mutate({{new_col}} := partA %>%
           select({{new_col}}) %>%
           dplyr::pull({{new_col}}) %>%
           gsub("[[:digit:]]+|[\r\n]|[[:punct:]]|([[:blank:]])\\1+",
                        "", .))
  partB <- partB %>%
    mutate(across(.cols = everything(),
                str_remove_all, "a")) %>%
    mutate(across(.cols = everything(),
                str_remove_all, ","))
  rm(TABLE)
  TABLE <- bind_cols(partA,
                     partB)
  TABLE <- TABLE %>%
  pivot_longer(cols = contains("20"), names_to = "Year", values_to = pivot_col)
  TABLE
}

I included the following line to help separate the tables.

---

Table11.1a

We then apply this function to the table, ridding the table of headings and ensuring some of our commons are correctly of numeric class.

dim(table11.1b)

[1] 21 18

table11.1b <- data_prep_settings(TABLE = table11.1b,
          old_col = "Setting Where Mental Health ServiceWas Received",
          new_col = "MHS_setting",
          pivot_col = "Percent")

table11.1b <- table11.1b %>%
  filter(MHS_setting != "General Medicine") %>%
  filter(MHS_setting != "Juvenile Justice") #Leading lines with no data

table11.1b <- table11.1b %>%
  mutate(across(c(Year, Percent), as.numeric))

We write a function to simplify this process for data that uses demographic groups as units of observation.

The function needs to:

• remove the last row of the table
• get rid of certain patterns
• transition the data to long format

data_prep_dem <- function(TABLE, old_col, new_col, pivot_col){
  TABLE <- TABLE[-dim(TABLE)[1],]
  TABLE <- TABLE %>%
  na_if("nc") %>%
  na_if("--") %>%
  na_if("") %>%
  na_if("*")
  TABLE <- TABLE %>%
    as_tibble() %>%
    rename({{new_col}} := {{old_col}})
  partA <- TABLE %>%
    dplyr::select({{new_col}})
  partB <- TABLE %>%
    dplyr::select(-{{new_col}})
  partA <- partA %>%
  mutate({{new_col}} := partA %>%
           dplyr::select({{new_col}}) %>%
           pull({{new_col}}) %>%
           gsub("[\r\n]|[[:punct:]]|([[:blank:]])\\1+",
                        "", .))
  partA <- partA %>%
  mutate({{new_col}} := dplyr::case_when(stringr::str_detect(!!base::as.name(new_col), pattern = "1") ~ base::paste("Age",
                                                        stringr::str_sub(!!base::as.name(new_col),
                                                                start = 1,
                                                                end =2),
                                                        stringr::str_sub(!!base::as.name(new_col),
                                                                start = 3,
                                                                end = 4),
                                                        sep="_"),
                                 TRUE ~ !!base::as.name(new_col)))
  partB <- partB %>%
    mutate(across(.cols = everything(),
                str_remove_all, "a")) %>%
    mutate(across(.cols = everything(),
                str_remove_all, ","))
  rm(TABLE)
  TABLE <- bind_cols(partA,
                     partB)
  TABLE <- TABLE %>%
  pivot_longer(cols = contains("20"), names_to = "Year", values_to = pivot_col)
  TABLE
}

I included the following line to help separate the tables.

---

Table11.2a

We use the produced function to wrangle the next pair of tables.

dim(table11.2a)

[1] 18 16

table11.2a <- data_prep_dem(TABLE = table11.2a,
          old_col = "Demographic Characteristic",
          new_col = "Demographic",
        pivot_col = "Number")

table11.2a %>%
  filter(!complete.cases(.)) %>%
  dplyr::group_by(Demographic) %>%
  tally()

# A tibble: 5 x 2
  Demographic                  n
  <chr>                    <int>
1 AGE GROUP                   15
2 AIAN                         1
3 GENDER                      15
4 HISPANIC ORIGIN AND RACE    15
5 NHOPI                       14

table11.2a <- table11.2a %>%
  filter(stats::complete.cases(.) | Demographic == "AIAN")

table11.2a <- table11.2a %>%
  mutate(across(c(Year, Number), as.numeric))

I included the following line to help separate the tables.

---

Table11.2b

dim(table11.2b)

[1] 18 16

table11.2b <- data_prep_dem(TABLE = table11.2b,
          old_col = "Demographic Characteristic",
          new_col = "Demographic",
          pivot_col = "Percent")

table11.2b %>%
  filter(!complete.cases(.)) %>%
  group_by(Demographic) %>%
  tally()

# A tibble: 5 x 2
  Demographic                  n
  <chr>                    <int>
1 AGE GROUP                   15
2 AIAN                         1
3 GENDER                      15
4 HISPANIC ORIGIN AND RACE    15
5 NHOPI                       14

table11.2b <- table11.2b %>%
  filter(complete.cases(.) | Demographic == "AIAN")

table11.2b <- table11.2b %>%
  mutate(across(c(Year, Percent), as.numeric))

We repeat this process for the remaining tables.

I included the following line to help separate the tables.

---

Table 11.3a

dim(table11.3a)

[1] 18 14

table11.3a <- data_prep_dem(TABLE = table11.3a,
          old_col = "Demographic Characteristic",
          new_col = "Demographic",
          pivot_col = "Number")

table11.3a %>%
  filter(!complete.cases(.)) %>%
  group_by(Demographic) %>%
  tally()

# A tibble: 5 x 2
  Demographic                  n
  <chr>                    <int>
1 AGE GROUP                   13
2 AIAN                         2
3 GENDER                      13
4 HISPANIC ORIGIN AND RACE    13
5 NHOPI                       13

table11.3a <- table11.3a %>%
  filter(complete.cases(.) | Demographic == "AIAN")

table11.3a <- table11.3a %>%
  mutate(across(c(Year, Number), as.numeric))

I included the following line to help separate the tables.

---

Table 11.3b

dim(table11.3b)

[1] 18 14

table11.3b <- data_prep_dem(TABLE = table11.3b,
          old_col = "Demographic Characteristic",
          new_col = "Demographic",
          pivot_col = "Percent")

table11.3b %>%
  filter(!complete.cases(.)) %>%
  group_by(Demographic) %>%
  tally()

# A tibble: 5 x 2
  Demographic                  n
  <chr>                    <int>
1 AGE GROUP                   13
2 AIAN                         2
3 GENDER                      13
4 HISPANIC ORIGIN AND RACE    13
5 NHOPI                       13

table11.3b <- table11.3b %>%
  filter(complete.cases(.) | Demographic == "AIAN")

table11.3b <- table11.3b %>%
  mutate(across(c(Year, Percent), as.numeric))

I included the following line to help separate the tables.

---

Table 11.4a

dim(table11.4a)

[1] 18 16

table11.4a <- data_prep_dem(TABLE = table11.4a,
          old_col = "Demographic Characteristic",
          new_col = "Demographic",
          pivot_col = "Number")

table11.4a %>%
  filter(!complete.cases(.)) %>%
  group_by(Demographic) %>%
  tally()

# A tibble: 7 x 2
  Demographic                  n
  <chr>                    <int>
1 AGE GROUP                   15
2 AIAN                        15
3 Asian                       15
4 GENDER                      15
5 HISPANIC ORIGIN AND RACE    15
6 NHOPI                       15
7 Two or More Races           12

table11.4a <- table11.4a %>%
  filter(complete.cases(.) | Demographic == "AIAN")

table11.4a <- table11.4a %>%
  mutate(across(c(Year, Number), as.numeric))

I included the following line to help separate the tables.

---

Table 11.4b

dim(table11.4b)

[1] 18 16

table11.4b <- data_prep_dem(TABLE = table11.4b,
          old_col = "Demographic Characteristic",
          new_col = "Demographic",
          pivot_col = "Percent")

table11.4b %>%
  filter(!complete.cases(.)) %>%
  group_by(Demographic) %>%
  tally()

# A tibble: 7 x 2
  Demographic                  n
  <chr>                    <int>
1 AGE GROUP                   15
2 AIAN                        15
3 Asian                       15
4 GENDER                      15
5 HISPANIC ORIGIN AND RACE    15
6 NHOPI                       15
7 Two or More Races           12

table11.4b <- table11.4b %>%
  filter(complete.cases(.) | Demographic == "AIAN")

table11.4b <- table11.4b %>%
  mutate(across(c(Year, Percent), as.numeric))

Now that we’ve wrangled the data, we can go ahead and proceed with our analysis.

Data Analysis

---

In this section, we only analyzed data from tables 2-4. Data from table 1 is very different than data from tables 2-4. For expediency, I did not include an example with data frome table 1. The following code, however, can easily be repurposed to accomplish that once a specific group has been identified to conduct the test on.

We would like to conduct a chi-squared test for independence.

To conduct this statistical test, we need to produce a 2x2 table.

The following code subsets the data we need and makes the necessary manipulations so that the units of observation are appropriate.

chi_square_11.2a <- table11.2a %>%
  filter(Year %in% c(2009, 2018)) %>%
  filter(Demographic %in% c("Male","Female")) %>%
  mutate(Number = Number * 1000)

The resulting object is still in long format.

chi_square_11.2a

# A tibble: 4 x 3
  Demographic  Year  Number
  <chr>       <dbl>   <dbl>
1 Male         2009  577000
2 Male         2018  946000
3 Female       2009 1377000
4 Female       2018 2537000

To conduct a chi-squared test for indepence we will need a contingency table.

A contingency table can be produced from data in long format by transforming the data to wide format and repurposing some values as row names.

chi_square_11.2a <- chi_square_11.2a %>%
  tidyr::pivot_wider(names_from = Year,
              names_prefix = "Year", 
              values_from = Number) %>%
  tibble::column_to_rownames("Demographic")

The final object should look like this.

chi_square_11.2a

       Year2009 Year2018
Male     577000   946000
Female  1377000  2537000

The chi-squared test for independence can be conducted using the stats::chisq.test() function.

stats::chisq.test(chi_square_11.2a)

    Pearson's Chi-squared test with Yates' continuity correction

data:  chi_square_11.2a
X-squared = 3482.7, df = 1, p-value < 2.2e-16

We can repeat this process for the remaining tables.

chi_square_11.3a <- table11.3a %>%
  filter(Year %in% c(2009, 2018)) %>%
  filter(Demographic %in% c("Male","Female")) %>%
  mutate(Number = Number * 1000)

chi_square_11.3a <- chi_square_11.3a %>%
  pivot_wider(names_from = Year,
              names_prefix = "Year", 
              values_from = Number) %>%
  column_to_rownames("Demographic")

chi_square_11.3a

       Year2009 Year2018
Male     391000   628000
Female  1013000  1795000

chisq.test(chi_square_11.3a)

    Pearson's Chi-squared test with Yates' continuity correction

data:  chi_square_11.3a
X-squared = 1696, df = 1, p-value < 2.2e-16

chi_square_11.4a <- table11.4a %>%
  filter(Year %in% c(2009, 2018)) %>%
  filter(Demographic %in% c("Male","Female")) %>%
  mutate(Number = Number * 1000)

chi_square_11.4a <- chi_square_11.4a %>%
  pivot_wider(names_from = Year,
              names_prefix = "Year", 
              values_from = Number) %>%
  column_to_rownames("Demographic")

chi_square_11.4a

       Year2009 Year2018
Male     168000   351000
Female   505000  1081000

chisq.test(chi_square_11.4a)

    Pearson's Chi-squared test with Yates' continuity correction

data:  chi_square_11.4a
X-squared = 50.256, df = 1, p-value = 1.349e-12

Data Visualization

---

This is the intentionally terrible plot that requires faceting.

table11.1b %>%
  ggplot2::ggplot(aes(x = Year, y = Percent, group = MHS_setting)) +
  ggplot2::geom_line() +
  ggplot2::scale_x_continuous(breaks = seq(2009, 2018, by=1),
                     labels = seq(2009, 2018, by=1),
                     limits = c(2009, 2018)) +
  ggplot2::labs(title = "Settings Where Mental Health Services Were Received in Past Year\namong Persons Aged 12 to 17",
       subtitle = "Percentages, 2002-2018")

The plots below need to be correctly faceted. Keep in mind that tables 11.2+ must be faceted by demographic group type and not by setting type.

table11.1b %>%
  ggplot(aes(x = Year, y = Percent, group = MHS_setting)) +
  geom_line() +
  scale_x_continuous(breaks = seq(2009, 2018, by=1),
                     labels = seq(2009, 2018, by=1),
                     limits = c(2009, 2018)) +
  labs(title = "Settings Where Mental Health Services Were Received in Past Year\namong Persons Aged 12 to 17",
       subtitle = "Percentages, 2002-2018")

table11.2b %>%
  ggplot(aes(x = Year, y = Percent, group = Demographic)) +
  geom_line() +
  scale_x_continuous(breaks = seq(2009, 2018, by=1),
                     labels = seq(2009, 2018, by=1),
                     limits = c(2009, 2018)) +
  labs(title = "Major Depressive Episode in Past Year\namong Persons Aged 12 to 17",
       subtitle = "By Demographic Characteristics, Percentages, 2004-2018")

table11.3b %>%
  ggplot(aes(x = Year, y = Percent, group = Demographic)) +
  geom_line() +
  scale_x_continuous(breaks = seq(2009, 2018, by=1),
                     labels = seq(2009, 2018, by=1),
                     limits = c(2009, 2018)) +
  labs(title = "Major Depressive Episode with Severe Impairment in Past Year\namong Persons Aged 12 to 17",
       subtitle = "By Demographic Characteristics: Percentages, 2006-2018")

table11.4b %>%
  ggplot(aes(x = Year, y = Percent, group = Demographic)) +
  geom_line() +
  scale_x_continuous(breaks = seq(2009, 2018, by=1),
                     labels = seq(2009, 2018, by=1),
                     limits = c(2009, 2018)) + 
  labs(title = "Receipt of Treatment for Depression in Past Year among\nPersons Aged 12 to 17 with Major Depressive Episode in Past Year",
       subtitle = "By Demographic Characteristics: Percentages, 2004-2018")

The plots created (after faceting properly) can be used to answer the questions listed at the beginning of the case study. After finalizing the plots, some time should be spent towards framing the visualizations in such a way to underscore how they were used to asnwer the question.

Summary

---

Suggested Homework

---

Additional Information

---

Helpful Links

This needs to be updated

guide for using GGally to create correlation plots

Terms and concepts covered:

Tidyverse
RStudio cheatsheets

Packages used in this case study:

Package	Use
here	to easily load and save data
tidyverse	R packages for data science
rvest	to scrape web pages

Acknowledgements

We would like to acknowledge Tamar Mendelson for assisting in framing the major direction of the case study.

We would also like to acknowledge the Bloomberg American Health Initiative for funding this work.

RA Notes

This is the motivating article for this case study. In this article, they web scrape to obtain the data they need.

Here is the Lieber Institute’s resource on web scrape

Here is a resouce the Lieber Institute source above referse to

Here as a good resource to learn how to web scrape

This is the set of tables we would like to consider